MCAS REPORTING ERROR 



10-1-11 



1 . The reporting error 



(a) The type of the error 

10 th grade students received incorrect scale scores (the scores that run 
from 200 to 280) because they were taken from the wrong "raw score to 
scale score conversion table" 



(b) The cause of error 



Based on an initial report from the contractor, Measured Progress, the use 

of the wrong conversion table was due to human error. 

The score reporting systems functioned properly, however, the staff 

person responsible for the selection of the conversion table used the wrong 

one. 



(c) The source of discovery 



ESE's Student Assessment Office received two calls that reported the 
error on Wednesday, 9-28-1 1 in the afternoon. One from the Principal of 
Westwood High School who reported that the reporting error had been 
brought to his attention by the parent of a student. The other call was 
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2. The scoring contract 



(a) The history of the contractor, including history w/ MA 

ESE's current five year contract with Measured Progress (M.P.) for 
MCAS runs from January 1, 2010 to December 3 1, 2014. The FY 201 1 
contract was for $25 million and will cost a similar amount this FY 2012. 
Overlapping with the MCAS contract, M.P. also won the contract for 
testing English Language Learners ("MEPA") which runs from July 1 , 
2007 to June 30, 2012; this contract cost 3.5 million in FY 201 1 and will 
incur similar costs in FY 2012. This will be the last year of the MEPA 
contract with M.P. ; we are transitioning to a different test contractor, 
MetiTech, the testing contractor for the multi-state consortium WIDA, in 
FY 2013. 



1 



M.P. held the prior five year contract from July 1, 2004 to June 30, 2009, 
with a six month extension to December 31, 2009. Initially in this 
contract period, M.P. subcontracted the high school testing program to 
CTB / McGraw-Hill. Significant problems with the subcontractor's 
performance led to the termination of this subcontract in 2006 and the high 
school testing program was subsumed by M.P. 

The previous MCAS contract was help by Harcourt which ran from 1999 
to 2004. 

(b) The terms of the contract 

Measured Progress is contracted to: 

i. Develop approximately 3,500 high quality selected response 
(multiple choice) and constructed response (e.g., for essays, math 
problem solving, etc.) every year of the contract for English 
Language Arts (ELA), Mathematics, and Science. [NOTE: 
Approximately 50% of items are released to the public in grades 3- 
8 and 100% of high school items are released each year.] 

ii. Assemble, deliver, score and report results each year for the 
following tests: 

• ELA Reading Comprehension, grades 3-8 and 10 
(Approximately 130 forms are administered in order to 
ensure high precision in equating results across years and to 
yield a bank of items for testing in the following year.) 

• ELA Composition, grades 4, 7 & 10 (approximately 56 
forms) 

• Mathematics, grades 3 -8 & 10 (approximately 160 forms) 

• Science, grades 5 & 8 (approximately 44 forms) 

• Separate high school Biology, Chemistry, Physics and 
Technology/Engineering tests, grades 9 & 10 
(approximately 36 forms) 

• NOTE: History/Social Science testing for grades 5 & 7, 
and US History in High School was suspended in 2009. 

iii. Ensure the highest technical quality of the tests (validity, reliability 
and comparability), equate results across forms and across testing 
years, and provide documented evidence of these characteristics. 

iv. The contract includes detailed specifications and timetables for all 
deliverables and penalties ("liquidated damages") for when these 
specifications and/or timelines are not met. 

3. Data quality assurance 

(a) The contractor's responsibilities 
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M.P. is responsible for primary quality control for an extensive array of 
data generation, transport and processing steps in the testing process. We 
will summarize these responsibilities Monday. 

M.P. is also responsible for a limited number of quality control redundant 
checks of the data. The genesis of the transition of this responsibility to 
M.P. is summarized in part "c" below. Under this area of responsibility, 
M.P. is responsible for the following: 

i. Validate the performance levels are consistent with the scaled cut 
scores. 

ii. Validate the performance levels are consistent with the theta cut 
scores 

iii. Recreate the scaled scores and achievement levels using the 
approved scaling constants and compare to psychometrics 

iv. Assign scaled scores and performance levels to students and 
compare to psychometrics. 

(b) The state's responsibilities 

ESE's Student Assessment Office is responsible for the following data 
quality responsibilities: 

• The accuracy of all student level data transmitted to the contractor 

• Contact information related to all districts, schools, and other 
entities that administer MCAS tests; in this case, the contractor is 
responsible for redundant quality control (QC) 

• Under the Department's direction, a second independent MCAS 
equating is performed on all grades by faculty of the U.Mass 
Research and Evaluation Methods Program. This analysis 
confirms that the raw score to scaled score tables proposed by the 
contractor were correctly calculated. The Department reviews the 
two equating analyses and approves the final lookup tables. 

• Stipulating the "business / reporting rules" that govern how all data 
is handled and reported. 

• Provide districts with a "discrepancy reporting window" during a 
full week in August when school and district leaders are expected 
to review all MCAS data and report any discrepancies; the 
Department reviews all discrepancy reports while M.P. is 
responsible for investigating the discrepancies that involve 
checking source materials 

ESE's Student Assessment Office is responsible for the following quality 
control redundant checks: 

• Every variable / "field" in the data files transmitted to ESE meet 
the data dictionary definition for that field 
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• Ensuring the accuracy of the list of students whose results must be 
invalidated due to security breaches 

• Computing the growth data, performed on the raw data provided 
by the contractor 

• Ensuring that the official data from the contractor is correctly 
transformed and loaded into the systems used to calculate AYP 
results and post them to the public website and to the secure 
reporting websites. 

(c) The recent contract change that limited shifted a limited number of quality 
control redundant checks from the state to the contractor ^s^ete 

In order to post ELA results before the close of school and to post 
preliminary Mathematics results in the first week of August, the 
Department had to reduce the time devoted to quality control redundant 
checks from one week to one day. The responsibility for producing the 
original reports of student results and then conducting the redundant 
quality control review was transferred from ESE's Student Assessment 
Office to the contractor. The contractor developed an independent 
protocol for this redundant review that was vetted and ultimately approved 
by the Student Assessment Office. [NOTE: Although the contractor has 
identified "human error" as the source of the scale score reporting 
problem, ESE has not yet been able to confirm that the contractor fully 
implemented the independent redundant review. This will be resolved in 
the coming days as we continue our investigation.] 

[NOTE: In addition to the data quality control responsibilities summarized above, 
the Student Assessment Office and contractor conduct several dozen quality 
control procedures on all other stages of the MCAS testing program from item 
development, to item acceptance, to the assembly of test forms, to the production 
of test forms, to the delivery, administration and recovery of test booklets and 
answer sheets. A summary of these additional QC responsibilities will be 
prepared and transmitted on Monday.] 

4. Steps taken by C.O.B. Friday September 30 th 

(a) Multiple analyses completed to determine the full scope of the problem; 
confirmed it is limited to 10 th grade mathematics 

(b) Multiple analyses completed to determine impact of the corrections to 
student scale scores, re, MCAS and AYP reports at the district, school and 
individual student levels 

(c) School districts and relevant associations informed with commitment to 
provide more detailed information next week; staff have already expedited 
transmittal of corrected student results at the request of some districts, e.g., 
Boston 
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(d) Board members and legislative leadership informed just prior to public 
release 

(e) Public release coordinated with EOE and Governor's Office 

(f) Contractor provided initial analysis of the cause and scope of errors made 
in reporting scale scores 

(g) Contractor instructed to produce corrected reports for parents, schools and 
districts; data files completed on Friday and production scheduled to begin 
on Saturday and completed by Monday morning. 

(h) These steps are being conducted at the sole expense of the contractor and 
will be QC'd by Student Assessment staff once completed and prior to 
forwarding the corrected reports to parents, districts and schools; this is 
separate from any liquidated damages ("penalties") that may be assessed 

(i) Contractor directed to complete the investigation and documentation of all 
actions or lack thereof that contributed to the erroneous reporting of scaled 
scores and provide a detailed report to Commissioner Chester early next 
week. 

(j) Contractor informed that the Department is analyzing grounds for 
implementing the liquidated damages provisions of the contract. 

5. Steps that will be taken and/or are being contemplated 

(a) Schools and districts will receive corrected MCAS and AYP data and 
reports next week and will forward corrected individual student MCAS 
reports to parents 

(b) Publicly facing web pages will be updated, e.g., "School and District 
Profiles" 

(c) ESE will complete its investigation, analyze the M.P. report, and 
determine appropriate corrective actions and consequences 

6. Commissioner's current thoughts, including a recommended response from the 
Governor 

/ haven 't drafted this anticipating that you would do so; let me know if you would 
like me to develop a draft. 
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